On population-based measures of agreement

Measuring agreement between qualified experts is commonly used to determine the effectiveness of a diagnostic procedure. Many methods are available for assessing agreement, including Cohen’s kappa, which is a very popular summary measure of agreement due to its appealingly simple usage and interpretation. However, it has been previously shown that a number of flaws exist in its usage, which can lead to biased estimates of agreement. Cohen’s kappa seriously underestimates the level of agreement between experts classifying items for a rare disease. In this paper we compare the traditional Cohen’s kappa with the true form of Cohen’s kappa when the measurement of agreement between many raters and items in an underlying diagnostic procedure is of interest for binary classifications. We demonstrate how an alternative newly developed kappa statistic based upon the class of generalized linear mixed models, that is interpretable in the same manner as Cohen’s kappa yet does not suffer from the same flaws can provide a more appropriate assessment of agreement in a population-based study. The form of the model-based statistic is examined for both the probit and logit scenarios. Simulation studies and an application to a cancer of the uterus study (Holmquist 1968) demonstrate the performance of the three different kappa statistics.